Dynamic Task Allocation for Crowdsourcing Settings
نویسندگان
چکیده
We consider the problem of optimal budget allocation for crowdsourcing problems, allocating users to tasks to maximize our final confidence in the crowdsourced answers. Such an optimized worker assignment method allows us to “boost” the efficacy of any popular crowdsourcing estimation algorithm. We consider a mutual information interpretation of the crowdsourcing problem, which leads to a stochastic subset selection problem with a submodular objective function. We present experimental simulation results which demonstrate the effectiveness of our dynamic task allocation method for achieving higher accuracy, possibly requiring fewer labels, as well as improving upon a previous method which is sensitive to the proportion of users to questions.
منابع مشابه
Adversarial Task Allocation
The problem of allocating tasks to workers is of long standing fundamental importance. Examples of this include the classical problem of assigning computing tasks to nodes in a distributed computing environment, as well as the more recent problem of crowdsourcing where a broad array of tasks are slated to be completed by human workers. Extensive research into this problem generally addresses im...
متن کاملDynamic Task Allocation Algorithm for Hiring Workers that Learn
The automation of hiring decisions is a well-studied topic in crowdsourcing. Existing hiring algorithms make a common assumption—that each worker has a level of task competence that is static and does not vary over time. In this work, we explore the question of how to hire workers who can learn over time. Using a medical time series classification task as a case study, we conducted experiments ...
متن کاملMitigating Herding in Hierarchical Crowdsourcing Networks
Hierarchical crowdsourcing networks (HCNs) provide a useful mechanism for social mobilization. However, spontaneous evolution of the complex resource allocation dynamics can lead to undesirable herding behaviours in which a small group of reputable workers are overloaded while leaving other workers idle. Existing herding control mechanisms designed for typical crowdsourcing systems are not effe...
متن کاملStatistical Decision Making for Budget Allocation in Crowdsourcing
In this short paper, we briefly describe some recent progress on statistical decision making for budget allocation in crowdsourcing. We address the budget allocation problem for two important labeling tasks in crowdsourcing: the categorization labeling task and pairwise ranking aggregation. We also show the connections between our work and the “proactive learning” framework proposed by Jaime Ca...
متن کاملStructuring Time through Participation in Micro-task Crowdsourcing: A Time Allocation Perspective
Small payments in micro-task crowdsourcing markets appear unreasonable compared with remunerations for regular work in the workplace, yet hundreds of thousands of micro-tasks are completed each day, and frequently by highly educated individuals. To explain this perplexing anomaly, we investigate individuals’ continuous participation in micro-task crowdsourcing from a time allocation perspective...
متن کامل